I truncate my table by executing a queryJob described here: https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries
"truncate table " + PROJECT_ID + "." + datasetName + "." + tableName;
i wait until the job finishes via
queryJob = queryJob.waitFor();
Truncate works fine.
Anyway, if i do an insert right after the truncate operation via
InsertAllResponse response = table.insert(rows);
it results in a
com.google.cloud.bigquery.BigQueryException: Table is truncated.
with following log:
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://www.googleapis.com/bigquery/v2/projects/[MYPROJECTID]/datasets/[MYDATASET]/tables/[MYTABLE]/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Table is truncated.",
"reason" : "notFound"
} ],
"message" : "Table is truncated.",
"status" : "NOT_FOUND"
}
Sometimes i have even to wait more than 5 Minutes between truncate and insert.
I would like to check if my table is still in the state "Table is truncated." periodically until this state is gone.
How can i request bigquery api in order to check if the table is ready for inserts?
How can i request bigquery api for get the status of the table?
Edit
example for reproduce can be found here
If a table is truncated while the streaming pipeline is still going on or performing a streaming insertion on a recently truncated table, you could receive some errors like mentioned in the question (Table is truncated), that's expected behavior. The metadata consistency mode for the InsertAll (very high QPS API) is eventually consistent, this means that when using the InsertAll API, it may get delayed table metadata and returns the failure like table truncated. The typical way to resolve this issue is to back-off and retry.
Currently, there is no option in the BigQuery API to check if the table is in truncated state or not.
Unfortunately the api does not (yet?) provide an endpoint to check the truncated state of the table.
In order to avoid this issue, one can use a load job via gc storage.
It looks like the load job respects this state, as i have no issues with truncate/load multiple times in a row.
public void load(String datasetName, String tableName, String sourceUri) throws InterruptedException {
Table table = getTable(datasetName, tableName);
Job job = table.load(FormatOptions.json(), sourceUri);
// Wait for the job to complete
Job completedJob = job.waitFor(RetryOption.initialRetryDelay(Duration.ofSeconds(1)),
RetryOption.totalTimeout(Duration.ofMinutes(3)));
if (completedJob != null && completedJob.getStatus().getError() == null) {
// Job completed successfully
} else {
// Handle error case
}
}